Skip to content

fix(debugger): rebuild breakpoint condition when a probe stops capturing - #10159

Draft
watson wants to merge 1 commit into
watson/DEBUG-6081/guardrail-metricsfrom
watson/DEBUG-6081/refresh-breakpoint-on-capture-disabled
Draft

fix(debugger): rebuild breakpoint condition when a probe stops capturing#10159
watson wants to merge 1 commit into
watson/DEBUG-6081/guardrail-metricsfrom
watson/DEBUG-6081/refresh-breakpoint-on-capture-disabled

Conversation

@watson

@watson watson commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds refreshBreakpoint(probe) to the breakpoints module, which replaces the breakpoint at a probe's location with one compiled from the current state of the probes attached to it, and calls it from the pause handler once a fatal capture error has permanently disabled snapshots or capture expressions for a probe.

Motivation

The breakpoint condition compiled for a location bakes in whether each probe produces snapshots. The sampler uses that flag to decide if a hit counts against the global snapshot rate limit and, since #10152, to classify a skipped hit as a snapshot or a log event. When the pause handler disabled capture for a probe the condition was never rebuilt, so such a probe kept eating into the global snapshot budget and its skipped hits kept being reported as snapshots, although every event it emitted from then on was a capture-less log event.

Additional Notes

The refresh runs after the main thread has been resumed and the results have been handed to the sender, under the same lock as the other breakpoint operations, and is a no-op for a probe that was removed in the meantime. All probes at a location share the breakpoint, so one refresh covers every probe disabled during the same pause.

The end-to-end test uses the large object safety threshold to trip the fatal error deterministically and checks that the following rate-limited hit is reported as events.skipped{event_type:log}; it fails without the fix (event_type:snapshot).

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Sep 4, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 5 Pipeline jobs failed

All Green | all-green

View more details · View in GitHub Actions

System Tests | main / parametric / parametric (1)

View more details · View in GitHub Actions

Failed to fetch archives due to 404 errors while trying to install dependencies during Docker image build.

System Tests | main / parametric / parametric (2)

View more details · View in GitHub Actions

Failed to fetch package 'libperl5.32_5.32.1-4+deb11u5_amd64.deb' during Docker image build. 404 error received.

View all 5 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 98.04% (+0.01%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 48fbe89 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 8.85 MB
Deduped: 9.51 MB
No deduping: 9.51 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.4.0 | 127.33 kB | 447.04 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.65%. Comparing base (12d404b) to head (48fbe89).

Additional details and impacted files
@@                         Coverage Diff                          @@
##           watson/DEBUG-6081/guardrail-metrics   #10159   +/-   ##
====================================================================
  Coverage                                98.65%   98.65%           
====================================================================
  Files                                     1007     1007           
  Lines                                   157793   157836   +43     
  Branches                                    74       74           
====================================================================
+ Hits                                    155670   155715   +45     
+ Misses                                    2123     2121    -2     
Flag Coverage Δ
ai-guard 65.04% <ø> (+0.03%) ⬆️
apm-capabilities 63.11% <ø> (ø)
apm-integrations 81.41% <ø> (ø)
appsec 58.71% <ø> (ø)
debugger 70.10% <100.00%> (+0.01%) ⬆️
instrumentation 53.72% <ø> (ø)
llmobs 79.00% <ø> (ø)
master-coverage 98.65% <100.00%> (+<0.01%) ⬆️
openfeature 66.38% <ø> (ø)
platform 69.31% <ø> (ø)
profiling 66.75% <ø> (ø)
serverless 65.71% <ø> (ø)
test-optimization 82.69% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@watson
watson force-pushed the watson/DEBUG-6081/refresh-breakpoint-on-capture-disabled branch from 8eee38a to 5f64639 Compare September 4, 2026 10:24
@pr-commenter

pr-commenter Bot commented Sep 4, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-09-07 10:51:18

Comparing candidate commit 48fbe89 in PR branch watson/DEBUG-6081/refresh-breakpoint-on-capture-disabled with baseline commit 12d404b in branch watson/DEBUG-6081/guardrail-metrics.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2307 metrics, 6 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable max_rss_usage [-27.219MB; +4.790MB] or [-11.823%; +2.080%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-24

  • unstable max_rss_usage [-27556.970KB; +28673.770KB] or [-8.299%; +8.635%]

scenario:encoders-0.4-immediate-flush-20

  • unstable max_rss_usage [-7.467MB; +9.992MB] or [-9.601%; +12.848%]

scenario:llmobs-encode-unicode-mixed-20

  • unstable max_rss_usage [-12.669MB; +6.238MB] or [-16.574%; +8.160%]

scenario:log-with-debug-20

  • unstable max_rss_usage [-7461.680KB; +7328.880KB] or [-6.575%; +6.458%]

scenario:plugin-graphql-long-with-depth-on-max-26

  • unstable max_rss_usage [-22.029MB; +30.004MB] or [-11.406%; +15.535%]

@watson
watson force-pushed the watson/DEBUG-6081/refresh-breakpoint-on-capture-disabled branch 3 times, most recently from 5644b43 to d1217f3 Compare September 4, 2026 15:41
@watson
watson force-pushed the watson/DEBUG-6081/refresh-breakpoint-on-capture-disabled branch from d1217f3 to 3c4c33e Compare September 6, 2026 09:05
The breakpoint condition compiled for a location bakes in whether each
probe produces snapshots. The probe sampler uses that flag to decide if
a hit counts against the global snapshot rate limit, and since the
guardrail metrics landed also to classify a skipped hit as a snapshot
or a log event.

When the pause handler hits a fatal capture error it permanently
disables snapshots or capture expressions for the probe, but the
breakpoint condition was never rebuilt. Such a probe kept eating into
the global snapshot budget and its skipped hits kept being reported as
snapshots, although every event it emitted from then on was a
capture-less log event.

Add `refreshBreakpoint()` to the breakpoints module, which replaces the
breakpoint at a probe's location with one compiled from the current
state of the probes attached to it, and call it from the pause handler
once the main thread has been resumed and the results are on their way.
@watson
watson force-pushed the watson/DEBUG-6081/refresh-breakpoint-on-capture-disabled branch from 3c4c33e to 48fbe89 Compare September 7, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant